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The proposed BASIC program -For smoothing NUWES data uses the Least- 
Squares Method to -Fit a polynomial to a 7-point data segment. It determines 
the best -fitting polynomial o-F order 1, 2, or 3 and establishes an estimate 

for the coordinate at the time in the center of the data segment. It also 
establishes a Fi gur e-o-F-Mer i t (FM) -For that estimate. 



This program is designed spec i -F i cal 1 y -for treatment of outliers and 
nissing points but it can also be used -For smoothing o-F other observations. 
It is intended to be used as the basic smoothing program for NUWES data and 
to provide a value of FM for each observation time for feedback to the 
Instrumentation Department. Values of the FM ' s also provide information 
For other users of the smoothed data on it's quality (i.e„, how well the 
smoothed values can be expected to represent the actual path of a vehicle.) 

This report is intended to supplement the Project Report entitled 
"A Figure of Merit for NUWES Data" . (Ref. 7) 



II. DATA PREPARATION 

It is assumed that the input data has been listed in the following 
format 



Time, Array, x comp, y comp, z comp. 

Each component (x,y,z) is smoothed separately and the Array identification 
is not used in the smoothing process so a separate file can be constructed 
for each component with the Array column replaced by a column for 
identification of Questionable Points (QP) , i.e. , for identification of 
hissing Points and Outliers. The identification numbers are specified with 
Vray A (6,5) which is presented in the next section of this report. The 
□reposed input format is 

Time, QP , XO 

Aihere XO is the observed value of x at the corresponding observation time. 

I wo preliminary steps must be taken before the smoothing can be performed. 
These are: 

A. Treatment of Missing Points. Since the program is designed to treat 
7-point data segments containing no missing points, the data must be 
screened and temporary values provided at each missing point. These 
:emporary values are established by taking a linear average of adjacent 
observational values. The following formulae are used to supply the 
:empor ar y val ues: 

1. For a single missing value at time t when there 
observed values xo(t-l) and xo(t.+ l) the temporary value is 

X O ( t ) « (XO(t-l) + XO<t+l))/2. 



are 



I 1 



observed 



For adjacent missing values xo(t) and xo(t+l) when there are 
values ko ( t — 1 ) and xo(t+2) the temporary values are 



x o < t ) 
xo (t+1 ) 



(2*xo ( t — 1 ) + xo<t+2))/3, 
(xo(t-l) + 2*xo(t+2> ) /3. 



3- For three adjacent missing values xo(t-l) , xo(t) , and xo(t+l) 
when there are observed values xo(t-2) and xo(t+2) the temporary values 
are 



: o ( t — 1 ) 
x o ( t ) 
:o (t+1 ) 



<3*xo (t-2) 
(xo(t-2) + 

( x o ( t —2 ) + 



+ xo(t+2) ) / 4 , 
xo (t+2) ) /2, 
3*xo (t+2) ) /4 . 



4, For -four adjacent missing values xo(t-l) , 
xo(t+l) , and xo(t+2) when there are observations xo(t-2) 
the temporary values are 



x o < t ) , 
and xo(t+3) 



xo(t-l) * 
x o<t) = 
xa(t+l) = 
xo ( t+2) = 



(4*xa (t-2) 
(3*xo (t-2) 
(2*xo ( t— 2) 

( x a ( t —2 ) + 



+ xo ( t+3) ) /b , 

+ 2*xa (t+3) ) /5, 
+ 3*xo (t+3) ) /5 , 
4*xa (t+3) ) /5. 






5- When there are more than -four adjacent missing values in a 7- 
point data segment the Least-Squares Method cannot be used to establish 
an estimate tor the midpoint of the segment. (It could still be 
possible, in some cases, to smooth at other points than the midpoint of 
the segment. This has not been considered as it would require 
considerable extension of the smoothing program and its desirability 
would be questionable.) 

B. Identification of Outliers. The procedure for identification of 
Outliers involves the use of Sequential Differences (Ref. 2 and Ref. 5). It 
will not be discussed in detail here. A Fourth Order difference of 
magnitude 50 or greater is considered an indicator that the corresponding 
value of xo is a potential outlier. When there are adjacent potential 
outliers, the one with the greatest magnitude of Fourth Order difference is 
identified as the actual outlier and assigned the appropriate number in the 
second column of Array A(6,5)« After smoothing this outlier, it might be 
prudent to recalculate the Sequential Differences with the outlier replaced 
by its smoothed value xe(t) to give assurance that the adjacent potential 
outliers are no longer identified as such. Cases where adjacent potential 
outliers will also be identified as outliers are, hopefully, rare and should 
be flagged for special treatment. 






III. THE SMOOTHING PROGRAM 



A. ARRAYS 

The program presented in this section was written in the BASIC Language 
-for use on an IBM PC. It uses 6 Arrays which will he discussed in the order 
that they appear in the program. 



f i rst 


Arr ay 


to be 


used is A ( 6 


,5) . It i 


s shown 


bel ow 








ARRAY 


A (6,5) 






i \ j 


0 


1 




3 


4 


5 




T 


QP 


XO 


XM 


XE 


RES 


0 


—3 


(0) 


xo(-3) 


x m ( —3 ) 


x e ( -3 ) 


r ( -3) 


1 


__ 


(-1 ) 


xo (-2) 


xm (-2) 


xe (-2) 


r (-2) 


o 


-1 


(-2) 


X o ( - 1 ) 


x m ( - 1 ) 


x e ( - 1 ) 


r (-1 ) 




0 


(-10) 


x o ( 0 ) 


x m ( 0 ) 


x © ( 0 ) 


r ( 0 ) 


4 


+ 1 


(-11) 


X O ( H- 1 ) 


' x m ( + 1 ) 


xe(+l) 


r ( •+• 1 ) 


5 


+2 


(-12) 


x o ( +2 ) 


xm (+2) 


x e ( +2 ) 


r (-i-2) 


6 


+3 


- 


x o ( +*3 ) 


x m ( +3 ) 


x e ( +3 ) 


r (+3) 



QUESTIONABLE POINTS <QP) 



O 


xo 


1 s 


-1 


X o 


i s 


o 


xo 


.i s 


-10 


X o 


i s 


-1 1 


xo 


i s 


-12 


xo 


i s 



legitimate observed value 
unscheduled missing point 
scheduled missing point 
identified as an outlier 
an unscheduled missing point and 
a scheduled missing point and an 



an outlier 
out 1 i er 



Smooth! 
estimate and 
column of Ar 
the observat 
rfi 1 1 always 
pr ovi des t 
cheduled mi 
future use. 
the temp or ar 
array will b 



ng of a 7— point data segment 
FM for the midpoint of the 
ray A is obtained by subtrac 
ion times of the data segmen 
be the same for all data seg 
h e i d en t i f i c at i on of t h e QP ' 
ssing points may be omitted 
The XQ's (column 3) are the 
y va 1 ues for the m l ss i n g pci 
e discussed as they appear i 



is performed to est ab 1 i sh t h e 
segment. The times in the first 
ting the time of this midpoint from 
t so that column 1 of this array 
rnents. The second column of Array 
s. A separate identification for 
but is included for potential 
:i. nput o b s e rved v a 1 ue s i n c: 1 u d i n g 
nts. The other columns of this 
n the smoothing program. 



Array T(5) provides the table values of the Student— T distribution 
;hat are needed to establish the Fi gure-of -Mer i t (FM) which accompanies the 
estimate >:e at the midpoint of the data segment. As discussed in Reference 
u 7 , the number 'infinity' cannot be used by a computer and hence it is 
arbitrarily replaced by the number 99.99 for T (0) . This value is large 
enough to produce a large value of FM and hence to indicate low quality of 
:he estimate it accompanies,, The argument j in T(j) ranges from 0 to 5 and 
s the appropriate number of ' degrees— of —freedom ' produced in Array 8(2,3). 
‘his will be discussed later. 



Array T (5) is shown below. 



ARRAY T<5> 



Q 





T ( j ) 99.99 


6.314 2.920 


tr “r 


2.132 2.105 






ARRAY B (2,3) 








i \ j 0 


1 2 








K 


DFK SDRK 


FMK 






0 1 


OF 1 SDR 1 


FM1 






1 2 


DF2 SDR2 


FM2 






o T 


DF3 SDR3 


FM3 




In this 


array K is the order of 


the pol ynomi al 


bei ng 


considered ( k ~ .1 for 


1 i near , 


k~2 for quadratic, and 


k“3 for cubic). 


DFK ’ i s 


the 'degrees of 


freedom ' 


available for fitting 


a polynomial of 


degree 


k and is established 


by first 


determining the number 


NS (el ement C (3 


) in Array C(8)) and then 


reduc i ng 


this number by the number of parameter 


s ( k + 1 ) 


in the polynomial. 


Thus 


NS = N - (number of QP 


' s in col umn A ( 1 


, J) ) 





= C<3) - (number of negative entries in column 

A ( 1 , j ) ) 

with, initially, 

C<3) = 7 = number of points in the data segment. 

Columns 2 and 3 are 

B(i,2) = estimate of the standard deviation of the noise 

in the observations (the xo's) when a polynomial 
of order k is fitted to the data segment 

an d 



B(i,3> ~ the Fi gure-of ~Mer i t for the estimate xe(0). 

ARRAY S<3) 

j 0 1 2 3 4 3 

S(j) x av ssx six s2x s3x s3x 1 

The element S(0)~xav is the average of the xo's. It is needed to 
produce the xm's (column 3 in Array A)., The other elements of S are used t 
calculate the values for the SDRK's and the FMK ' s in Arr£*y B. The element 
of this array must be zeroed at the start of each smoothing cycle (line 
of the program) since they contain cumulations of product of the data. 



N w 



ARRAY D<3> 



J 

D ( j ) 



0 

bkO 



1 

bkl 



b k 2 



b k 3 



This array 
selected to -fit 
i ter at i on - 



is -For the coefficients of the polynomial with the order k 
the data segment- Its elements must be zeroed before each 



ARRAY C ( 8 ) 

j 0 1 2 3 4 5 6 7 3 

C(j) T QP XD NS ITER K XE RES FN 

This array provides the -Final output o-F the smoothing program. In 
addition to the time (C(0)) -for which the smoothing was performed and the 
observed or temporary value o-f xo <C<2>) at that time, it identifies the 
nature o-f the QP <C<1>> and the number of legitimate observations in tahe 
data segment- It reports the number of iterations of the smoothing proems 
(C(4)) and the order of the polynomial used to fit the data- It contains 
<the smoothed value XE <C<6)) and the accompanying value of FM(C(8))- 
Finally, it includes the residual error (C(7)) which is the difference 
jbetween the original value of xo (C(2>> and the estimated value (C<6)). 

Mote that this residual error can differ from the value of the residual 
error r (0) in cell A (3,5) of Array A when smoothing is iterated- For 
lexarnple, if A (3,1) (and C(l)) has the value -10, then the residual in C(7) 
is the difference between the outlier value of xo in C(2) and the estimated 
value xe in C(6) whereas the r esi dual error in A<3,5) is the? difference 
Detween the estimated value xe(0) in cell A(3,4) after the last iteration 
and the value of xo(0) in cell A<3,2) which is the value of xe(0) in cell 
A(3,4) from the preceding iteration- 



B. THE PROGRAM 



A listing of the BASIC smoothing program follows. 

BASIC PROGRAM LS7T3.DOC 

05 REM LS7T3.DOC . 17 Feb., 1985 

10 DIM A ( 6 , 5 ) , B ( 2 , 3 ) , C<8>, D<3>, D(5), T<5) 

20 DATA -3, -2, -1, 0, 1, 2, 3 

30 FOR i =0 TO 6 
40 READ A ( i , 0 ) 

50 NEXT i 

55 REM STUDENT T 

60 DATA 99.99, 6.314, 2.920, 2.353, 2.132, 2.105 
70 FOR j=0 TO 5 
80 READ T ( j > 

90 NEXT j 
100 C (3) = 7 

105 REM ENTER SMOOTHING TIME 
110 C(0> = 2160 

115 REM ENTER QUESTIONABLE POINTS (QP's) 

120 DATA -10, 0, 0, -2, 0, 0, 0 
130 FOR i =0 TO 6 
140 READ A < i , 1 ) 

150 NEXT i 

160 C(l) = A (3,1) 

165 REM ENTER OBSERVED XO 

170 DATA 33238.6, 33267.7, 33313.5, 33374.2, 33434.8, 33510.5, 

33592.5 

180 FOR i =0 TO 6 
190 READ A ( i , 2 ) 

200 NEXT i 

210 C ( 2 ) = A (3,2) 

215 REM DETERMINE NS 
220 FOR i=0 TO 6 

230 IF A ( i , 1 ) < 0 THEN C(3)=C(3)-1 ELSE C(3)=C(3) 

240 NEXT i 

245 REM CHECK IF SMOOTHING POSSIBLE 

250 IF C ( 3 ) < 3 THEN GO TO 260 ELSE GO TO 290 

260 C ( 4 ) =0 : C(5)=0 C(6)=C(2) 

270 C ( 7 ) =0 : C(8)=T(0) 

280 GO TO 1050 
290 C ( 4 ) =0 

295 REM START SMOOTHING 

300 C ( 4 ) =C ( 4 ) + 1 'ITERATION 

305 REM ZERO S AND D CELLS 

310 FOR j =0 TO 5 

320 S ( j ) =0 

330 NEXT j 

340 FOR j =0 TO 3 

350 D ( j ) =0 

360 NEXT j 

363 REM PRELIMINARY CALCULATIONS 
366 REM CALCULATE XAV AND XM's 
370 FOR i =0 TO 6 
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380 S <0)=S (0) +A ( 1 ,2) 

390 NEXT i 

400 S ( 0 ) =S ( 0 ) / 7 ' XAO 

410 FOR i =0 TO 6 

420 A (i ,3) =A ( i ,2) -S (0) ' XM 

430 NEXT i 

435 REM CALCULATE SUMS OF PRODUCTS 
440 FOR i =0 TO 6 

450 S ( 1 ) =S ( 1 ) +A ( i , 3 ) * A ( i , 0 ) 'SSX 

460 S(2)=S<2) +A(i ,3 ) *A < i ,0) 'SIX 

470 S ( 3 ) =S ( 3 ) + A ( i , 3 ) * ( A ( i , 0 ) ""2 ) ' S2X 

480 S ( 4 ) =S ( 4 ) + A ( i , 3 ) * ( A ( i , 0 ) •"•3 ) ' S3X 

490 NEXT i 

500 S<5)=(28*S<4)-196*S<2) ) /28 'S3X1 

505 REM CALCULATE FM1 

510 B < 0 , 1 ) -C ( 3 ) -2 ' DF 1 

520 SSR1= <23*S ( 1 ) -S<2) A 2> /28 

530 B ( 0 , 2 ) =SOR ( SSR 1 / B ( 0 , 1 ) ) 'SDR1 

540 B ( 0 , 3 ) =B ( 0 , 2 ) *T ( B ( 0 , 1 ) ) /SQR ( C ( 3 ) ) ' FM 1 

545 REM CHECK QUADRATIC FIT 

550 B ( 1 , 1 ) =C (3) -3 ' DF2 

560 IF B ( 1 , 1 ) < 1 THEN GO TO 690 ELSE GO TO 570 
565 REM CALCULATE FM2 
570 SSR2= <84*SSR1-S (3) -•2) /84 

580 B ( 1 ,2) =SQR ( SSR2/B (1,1)) ' SDR2 

590 B (1 ,3>=B ( 1 ,2) *T (B ( 1 , 1 ) ) /SQR (C(3> ' FM2 

595 REM CHECK CUBIC FIT 



600 

610 

620 

630 

635 

640 

650 

660 

665 

670 

680 

685 

690 

695 

700 

705 

710 

720 

730 

740 

750 

760 

765 

770 



B (2 , 1 ) =C (3) -4 ' DF3 

IF B ( 2 , 1 > < 1 THEN GO TO 620 ELSE GO TO 640 
B (2 , 2) =0 : B ( 2 , 3 ) =T ( 0 ) 

GO TO 670 



REM CALCULATE FM3 

SSR3= <216*SSR2— S (5) ""2) /216 

B (2 , 2) =SQR (SSR3/B (2, 1 ) ) ' SDR3 

8<2,3)=B(2,2)*T(B(2, 1) ) /SQR(C(3) ) ' FM3 

REM SELECT POLYNOMIAL 

IF B (2 , 3) <B (1,3) AND B (2 , 3) <B <0 ,3) THEN GO TO 850 
ELSE GO TO 680 



IF B( 1,3)<B (0,3) THEM GO TO 770 ELSE GO 
REM FIT LINEAR 
C (5) =1 : C (8) =B (O ,3) 

REM ESTABLISH COEFFICIENTS 
D ( 1 ) -S (2) /2S 
REM DETERMINE XE, RES 
FOR i =0 TO 6 
A ( i , 4 > =S ( 0 > +D < 1 > * A < i , 0 ) 

A(i ,5)=A(i ,2)-A(i ,4) 

NEXT i 

C ( 6 ) —A ( 3 , 4 ) : C (7) =C (2) -C (6) 

GO TO 930 

REM FIT QUADRATIC 

C (5) =2 : CCS) =B (1,3) 



TO 690 
k = 1 FM=FM1 
b2 1 



x e 
res 



k=: 






FM=FM2 



775 REM ESTABLISH COEFFICIENTS 

780 D(2)=S(3)/84 : D(l>= S(2)/28 : D (0) =-4*D (2) 

785 REM DETERMINE XE, RES 
790 FOR i =0 TO 6 

800 A ( i , 4 ) =S < 0 ) +D (0 ) +D ( 1 ) *A ( i ,0) +D <2 ) * < A ( i ,0> ""2> ' xe 

810 A ( i , 5) =A ( i , 2) -A ( i , 4) 'res 

820 NEXT i 

830 C(6)=A<3,4) : C (7) =C <2> -C (6) 

340 GO TO 930 
845 REM FIT CUBIC 

850 C ( 5 ) =3 : C(8)=B(2,3> 'k=3 FM=FM3 

855 REM ESTABLISH COEFFICIENTS 

860 D(3)=S(5) /216 : D(2)=S(3)/84 

870 D(1>=(S<2>-196*D<3> ) /28 : D(0)=-4*D(2> 

875 REM DETERMINE XE RES 
830 FOR i =0 TO 6 

890 A ( i , 4 > =S ( 0 ) +D < 0 > +D < 1 > * A < i , 0 > +D < 2 ) * < A < i , 0 > ) +D ( 3) * ( A < i , 0 ) 

900 A(i ,5)=A(i ,2)-A(i ,4) 'res 

910 NEXT i 

920 C (6) =A (3,4) : C (7) =C (2) -C (6) 

925 REM TERMINATE IF NOT QP 

930 IF A ( 3 , 1 ) =0 THEN GO TO 1050 ELSE GO TO 940 
935 REM CHECK RESIDUALS 
940 FOR i =0 TO 6 

950 IF A ( 1 , i ) < 0 AND ABS ( A ( i ,5) ) >1 THEN GO TO 980 ELSE GO TO 960 
960 NEXT i 
970 GO TO 1050 

980 IF C ( 4) >9 THEN GO TO 1050 ELSE GO TO 990 
990 FOR i=0 TO 6 

1000 IF A ( i , 1 ) < 0 THEN A ( i , 2 ) =A ( i , 4) ELSE A < i , 2) =A ( i , 2) 

1010 NEXT i 
1020 GO TO 300 
1045 REM TERMINATE 
1050 FOR j=0 TO 8 
1060 LPRINT C ( j ) ; 

1070 NEXT j 
1030 LPRINT 
1090 STOP 
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C. DISCUSSION OF THE PROGRAM 



The smoothing program can now be discussed in same detail. 



STEP 1. Enter the segment data. 



Before starting the smoothing the pertinent information on the data 
segment must be entered- This includes the time (program line 110), the 
QP's (line 120), and the observed xo's (line 170). 



STEP 2. Determine NS. 



This is the number of legitimate observations in the segment. 
<lines220-240> 

STEP 3 « Ghee k i f smoot hing possible. 

If NS<3, then even linear polynomial cannot be fitted to the data 
segment by the Least-Squares Method. (lines 250-280) If no fit is 
possible, then terminate (STEP 15). 

STEP 4. Zero iteration counter and cells in Arrays S and D. 

S cells cumulate data products and D cells contain parameter 
coef f i ci ents. ( 1 i nes 290-360) 

STEP 5. Calculate xav (S(0)) and xm's (A(i,3). 

xav is the average of the xo's and the xm's are the differences 
;<it)( j)=xo( j)-xav. This reduces computational round-off error. 

( 1 i nes 370-430) 



STEP 6. Calculate other S(j) 's. 

These are sums of products of data values, (lines 440-500) 

STEP 7. Determine capability of linear fit. 

Establish DPI (B(0,1>>, SDR1 (B(0,2)>, and FM’l (B(0,3)). 

(lines 510-540) 

STEP 8. Check quadratic fit. 

If NS < 4 then DF2 < 1 and no quadratic fit is possible. Otherwise 

establish SDR2 and FM2. (lines 550-590) 

STEP 9. Check cubic fit. 

If NS < 5 then DF3 < 1 and no cubic fit is possible. Otherwise 

establish SDR3 and FM3. (lines 600—660) 

STEP 10. Select polynomial for fitting data segment. 



9 



If FM3 < FN2 and FM3 < FM1 then select cubic, set k = CCS) - 3 and fi 
cubic in STEP 13. Otherwise, if FM2 < FMl then select quadratic, set 
k = C(5) = 2 and fit quadratic in STEP 12. Finally, if neither of the abovi 
then select linear, set k - 0(5) =1 and fit linear in STEP 11. 

( 1 i nes 670-680 ) . 

STEP 11. Fit linear polynomial. 

Establish linear coefficients (D(i) 's) and calculate xe(i) ' s and 
r(i) 's, also xe(0) in cell C(6) and res in cell C(7). (lines 690-750) 

STEP 12. Fit cubic polynomial. 

Establish quadratic coefficients and calculate estimates and residuals 
( 1 i nes 770-830 ) 

STEP 13. Fit cubic polynomial. 

Establish cubic coefficients and calculate estimates and residuals. 

( 1 i nes 850-920) 

STEP 14. Check QP's. 

If xo(0) is legitimate observation (A(3,l) - 0) then terminate (STEP 

15)., Otherwise check number of iterations (C(4)). If C(4) = 10 then 

terminate. Finally, check residuals at times where A(i,l)<0 (at missing 
points and outliers). If any of these are greater than unity in magnitude 
then repeat the smoothing (STEP 4). If none of them are then terminate, 
(lines 930-1020) 

STEP 15. Terminate. 

Print Array C and STOP, (lines 1050-1090) 
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IV. APPLICATION 



The example used to illustrate this smoothing program is the sample 
selected consists of 40 points -from times t = 2 121 to t=2160. Additional 
points at both ends of the sample are required in order to smooth at these 
times. Appropriate inputs for this sample are given in the first four 
columns of Table 1. It is convenient to have the contents of Figure 1 on 
the monitor when performing the data smoothing. Inputs can then be changed 
easily by making the appropriate changes in lines 110, 120, and 170. This is 

particularly convenient when smoothing values other than outliers or missing 
values. The computer printout of the results of smoothing this sample are 
shown in Figure 2 and summarized in the last three columns of Table 1. 

It is of some interest to compare the results in Table 1, 
established using the BASIC Program LS7T3.BAS on an IBM PC, with the results 
given in Table 1 of Reference 7 which were established using a Texas 
Instruments calculator (TI-59). The differences can be attributed to the 
differences in the number of places used in the computations. 

Operation of the smoothing program can be examined in more detail by 
revising the program to version LS7T4.BAS. Application of this revised 
Drogram to the treatment of the missing point at t : =2160 is shown in Figure 
3a. This figure presents the elements of the other arrays as well as Array 
3. Several aspects of the smoothing of this missing point are discussed 
Del ow . 

1. The initial value value for xo(0) at time t-2160 which is entered 

in A(3,2) is the temporary value xo (0) ~ (xo (-1 ) +xo ( + 1 ) ) /2=*33374. 2. 

This value also appears in C(2> of the output. The value shown in 
A(3,2) in the printout of Array A is the value of xe(0) in A(3,4) 
from the next to last iteration. 

2. The initial value of xo(~3) entered in cell A (0,2) is the smoothed 
value for the outlier at t-2157 which was established when that 
outlier was smoothed. Again, the value in A (0,2) in the printout 
of Array A is the value of xe(-3) in A (0,4) from the next to last 

i ter at i on . 

3. The residual error r (0) =0.184 in A (3,5) is the ditf e?rence between 

the values of xo(0) in A(3,2) and xe(0) in A(3,4) whereas the 

residual error in C(7) is the difference between the temporary 
value of xo(0) initially entered in A<3,2) and in C(2) and the 
smoothed value xe(0) in A (3,4) and in C(6). 

4. In each iteration of the smoothing process the value of xo(0) in 
A (3, 2) was replaced by the estimate xe(0) in A (3, 4). The 
difference r (0) shown in A(3,5) is r (0)-xo(0)-xe(0) which is 
displayed in Array A as A (3 , 5) =A (3 , 2) -A (3 , 4) =0. 2 . The residual 
error shown in C(7) is the difference between the origninal value 
in A (3,2) which is saved in C(2) and the final estimate xe(0> in 

C (6) and in A(3,4) so that it is the difference between the initial 
temporary value and the final smoothed value, 



C ( 7 ) =C ( 2 ) — C ( 6 ) =4 - 8 . 

5- In the same way, the residual error in A(-3,5) is the difference 
A ( -3 , 5 ) =A (3,2) -A (3,4) =0 . 1 

whereas the -fitting error at t=2157 is the difference between the 
initial value in A(-3,2) which is the previously smoothed value 
(and also the input value) and the smoothed value o+ xe(-3) in A( 
3,2) so that 

RES ( -3 ) =33238 . 6-33234 . 5=4 . 1 . 



The fina 
segment 
with the 
quest i on 
smoothed 



1 est i mate x e ( -3 ) in A ( -3 , 4 ) is 
centered at t=2160 whereas the 
7-point data segment centered 
of : whether previously smoothed 
when smoothing other points. 



consistent with the 7-point data 
initial value xo(-3) is consistent 
at t =2157. This brings up the 
values should be iteratively 
This warrants -further investigation 



12 



TREATMENT OF MISSING POINT AT T 



2160 



INPUT OP'S 
-10 0 0 -2 0 
INPUT XO'S 
33238 . 6 33267.7 



0 0 



OOO 1 J. J 



33374 . 2 



33434. S 



335 1 0 - 5 



■TTtrno cr 
•JOjVjl. * 



ARRAY C (OUTPUT) 
2160 —2 33374.: 



33369.4 4.804688 1.941253 



ARRAY A 
-3 - 1 0 



m n 

~1 

0 

1 



(.) 

0 

jU. 

0 

0 

0 



33234.56 
33267 . 7 
>33 13.5 
33369.58 
3434.8 

335 1 0 . 5 

53592.5 



-154. 
- 121 . 
-75. 
— 19. 

45. 
121 . 
203 . 



457 

3203 

51953 

44141 

78125 

4805 

4805 



33234.5 
33267.94 
33313.25 
33369 . 4 
33435. 35 
335 1 0 . 1 

33592.6 



0625 
, 2421875 
25 

, 1835938 
5507813 
, 4023438 
, 1015625 



ARRAY 8 

0 3 26.00435 

0 2 1 . 84462 

0 1 . 687484 



27.36421 
2.40882: 
1 . 94125:: 



ARRAY S 

133389.02 102914.5 1680.715 412.1133 11728.02 -36.98438 

ARRAY D 

-19.62444 61.2241 4.906111 -.171224 



1 3 



TABLE 1 

Smoothing Sample NWS2AX1 



T 


COLL 


xo 


QP 


XE 


RES 


FM 


2117 


3 


33533.2 


0 


— 




— 


2118 


3 


33567 . 7 


0 


- 


- 


- 


2119 


3 


33603 . 6 


0 


- 


- 


- 


2120 


m 


(33634. 1 ) 


0 


33634 . 9 


( -0 . 8 ) 


1 .51 


2121 


3 


33664 . 4 


0 


33665.3 


-0. 7 


1.53 


2 1 r > r ? 


O 


33695 . 0 


0 


33693. 5 


1 . 5 


1 .41 


9 19“T 


3 


337 18.5 


0 


33720.9 


-2.4 


2.41 


2124 


3 


33745.8 


0 


33744.6 


1 . 2 


1 . 64 


2125 


3 


33767. 1 


0 


33764.6 


2.6 


3. 02 


2126 


3 


33780.7 


0 


33783. 7 


-3 . 0 


3.13 


2127 


3 


33798.0 


0 


33799.7 


-1 . 7 


3.47 


2128 


m 


( 338 10.9) 




338 16.3 


(-5.4) 


5. 17 


2129 


3 


33823 . 7 


0 


33828. 1 


-4.4 


4.73 


2130 


3 


33827.3 


0 


33821.6 


5.7 


8.35 


2131 




33794.2 


0 


33786. 6 


7.6 


1 1 . 23 


O 1 TO 
_r_ 1 •_> 


— .r 

S- 1 


33726. 1 


0 


33724. 1 


2 . 1 


2.46 


2133 


0 


33637 . 7 


0 


33640. 1 


- 2.4 


2.62 


2 1 34 


12 


33556.5 


0 


33556.9 


- 0.4 


5 . 07 


2135 


12 


33486.6 


0 


33490 . 0 


- 3.4 


3 . 45 


2136 


m 


( 33466 . 5 ) 


-12 


33452. 3 


( 14 . 2 ) 


4.61 


2137 


3 


33446 « 5 


0 


33451.2 


- 4.7 


5 . 16 


2 1 38 




33485. 1 


0 


33489.9 


- 4.8 


8.82 


2139 


3 


33559.3 


0 


33560.9 


-1 . 6 


3 . 44 


2140 




33650.2 


0 


33649.8 


0.4 


1 . 67 


2141 


0 


33738. 5 


0 


33734 . 5 


4.0 


5. 50 


2142 


3 


33799 . 0 


0 


33795.6 


3 . 4 


7.38 


2143 


.3 


33825.8 


0 


33823. 1 


2 . 7 


2.48 


2144 


m 


( 33798 . 7 ) 


-12 


33813. 0 


(- 14 . 3 ) 


2.93 


2145 


3 


33771 . 6 


0 


33767.4 


4.2 


6.62 


2146 


3 


33698 . 3 


0 


33696. 5 


1 . 8 


5.11 


2147 


3 


33607 . 7 


0 


33614.6 


- 6.9 


9 . 35 


2148 


12 


33528.5 


0 


33530 . 0 


- 1.3 


3.59 


2149 


12 


33455.2 


0 


33451 . 7 


3 * . 5 


3.84 


2150 


3 


33381.2 


0 


33383. 4 


. x. 


2.74 


2151 


3 


T T T O T 

•—» Xm . VJ 


0 


•33325 . 2 


- 1.7 


“T ~J 


2 1 52 


rn 


( 33285 . 1 ) 





33277.9 


( 7 . 2 ) 


2.36 


2 1 53 


.3 


33246.6 


0 


33243.5 


3 . 1 


3 . 02 


2154 


3 


33219.5 


0 


33222. 1 


- 2.6 


4 . 60 


2155 




332 12.5 


0 


332 14.0 


- 1.5 


4.64 


2156 


3 


33221 . 0 


0 


33218. 8 


0 0 


2 . 63 


2157 


3 


T "■* 0 y t 


-10 


33238.6 


34.9 


2 . 54 


2158 


3 


33267 . 7 


0 


33269.7 


-2 . 0 


2.51 


2159 


3 


33313.5 


0 


33313. 1 


0.4 


3 . 27 


2 1 60 


m 


( 33374 . 2 ) 


0 


33369 . 4 


( 4 . 8 ) 


1 . 94 


2161 


3 


33434 . 8 


0 


- 


- 


- 


2162 


3 


335 10.5 


0 


- 


- 


- 


2163 


3 


33592.5 


0 


- 


•- 


- 
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FIGURE 1 

BASIC SCREEN FOR SMOOTHING DATA 

LPRINT “SAMPLE NWS2AX1" 

..PRINT “OUTLIERS" 

..1ST 110 
110 C ( 0 ) =2 1 60 
:> 



-1ST 120 

120 DATA -10, 0, 0, -2, 0, 0, O 
I) 

1ST 170 

170 DATA 33238.6, 33267.7, 33313.5, 33374.2, 33434.8, 33510.5, 33592.5 
) 



3UN 



.PRINT “MISSING POINTS" 
.PRINT “OTHER POINTS" 
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TREATMENT OF SAMPLE NWS2A1X USING LS7T3 - BAS 



OUTLIERS 



2136 


-12 


33466.5 


6 


4 


3 


33452.28 


14.22266 


4. 610405 


2144 


— 12 


33709.7 


6 


4 


3 


33812.96 


-14.25781 


2. 9266. 7< 


2157 


-10 


77077 cr 


5 


4 




33238.63 


34.875 


2.541132 


MISSING POINTS 
2120 -2 


33634. 1 


6 


1 


00 


33634.88 


-.7734375 


1 . 475461 


2128 


jU 


338 1 0 . 0 


6 




. 


33816.34 


-5.4375 


5. 16895 


2152 


Xm 


33285. 1 


6 


.3 




33277.93 


7. 175782 


2. 356863 


2 1 60 




33374 . 2 


5 


-i 




33369.4 


4.804688 


1.941253 


OTHER POINTS 


2121 


0 


33664.6 


6 


1 




33665. 28 


-. 6757813 


1.526676 


2 1 22 


0 


33695 


6 


1 


2 


33693.54 


1 . 460938 


1 . 408332 


2123 


0 


33718. 5 


6 


1 




33720.87 


-2.371094 


2.412839 


2 1 24 


0 


33745 . 0 


7 


1 


'•0 

a. 


33744.58 


1 . 226563 


1 . 636923 


2125 


0 


33767. 1 


6 


1 


n 

a. 


33764.55 


2.550781 


3 . 020508 


2126 


0 


33780 . 7 


6 


1 




33783.69 


-2.988281 


3. 133061 


2127 


0 


33798 


6 


1 




33799.72 


-1 . 722656 


3.467283 


2129 


0 


33823. 7 


6 


1 


3 


33828. 12 


-4.421875 


3.467233 


2130 


0 


33827.3 


6 


1 


-;i; 


33821 . 64 


5 . 664063 


8.346241 


2131 


0 


33794.2 


6 


1 


X. 


33786.6 


7.605469 


1 1 . 22599 


2132 


0 


33726. 1 


7 


1 




33724.05 


2. 050781 


2.45981 1 


2 1 33 


o 


33637 . 7 


6 


1 




33640. 13 


-2.433594 


2. 61768 


2134 


0 


33556.5 


6 


1 


3 


33556.91 


-.40625 


5.074126 


2135 


o 


33486 . 6 


6 


1 


3 


33489.96 


-3.359375 


3.445967 


2137 


0 


33446.5 


6 


1 




33451 . 15 


-4.652344 


5. 162057 


2 1 38 


0 


33485. 1 


6 


1 


3 


33489.95 


-4. 792969 


3.815254 


2139 


0 


33559 . 3 


6 


1 


3 


33560.92 


-1.617138 


3. 443657 


2 1 40 


o 


33650.2 


7 


1 


3 


33649.8 


. 3984375 


1 . 67 1507 


2141 


0 


33738.5 


6 


1 


3 


33734.47 


4 . 03 1 2575 


5. 498794 


2142 


0 


33799 


6 


1 


3 


33795.56 


3 . 4453 1 3 


7. 332694 


2143 


0 


33825.8 


6 


1 


oo 


33823. 13 


2.671875 


2.47908 


2145 


0 


33771 . 6 


6 


1 


3. 


33767.36 


4.242188 


6 . 6 1 6293 


2146 


0 


33698 . 3 


6 


1 


3 


33696.5 


1 . 80078 1 


5. 109362 


2147 


0 


33607 . 7 


6 


1 


3 


336 1 4.61 


-6.910157 


9.347524 


2148 


0 


33528.5 


7 


1 


3 


33529.77 


-1.269531 


3.590484 


2149 


0 


33455.2 


6 


1 


'*0 


3345 1 . 7 1 


3.492188 


3.333284 


2 1 50 


0 


3338 1 . 2 


6 


1 


“f 


333 83.39 


-2. 1875 


2,. 737984 


2151 


0 


33323 . 5 


6 


1 


00 


33325. 17 


— 1 . 664063 


3 . 27 1 4 1 3 


2153 


0 


33246 . 6 


6 


1 


o? 


33243. 54 


3. 066406 


3 . 0 1 6 1 05 


2154 


0 


332 19.5 


cr 


1 


'0 


33222 . 07 


-2.570313 


4 , 597027 


2155 


0 


770 1 kt 

.c» L ,c . U 


5 


1 


0> 


33214.04 


-1 . 542969 


4.636977 


2156 


0 


"7, *7? '■? j. 


6 


1 


00 


33218.8 


2. 199219 


2.634624 


2150 


o 


33267.7 


5 


1 


00 


33269.67 


-1 . 964844 


2.507949 


2159 


0 


333 13.5 


5 


1 


•-r 


333 1 3 . 05 


. 4492188 


3.274325 
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CONCLUSIONS AND RECOMMENDATIONS 



vC 



The BASIC program LS7T3.BAS presented in this report uses the Least- 
Squares Method to select a polynomial of order 1, 2, or 3 to fit 7-point 
segments of data- It establishes an estimate for the actual value at the 
nidpoint of the data segment and a Figure of Merit (FM) to provide an 
indication of the quality of that estimate. 

A modified version of this program (LS7T4.BAS) is available for a more 
detailed examination of the application of the program to any 7-point data 
segment . 



It is recommended that this program be used for smoothing NUWES 3-D 
data. It is also recommended that this program be transferred to a 
spreadsheet format for use with LOTUS-123 with the intent of using macros to 
automate its application to the extent possible. 

Possible modification of this program to change the treatment of 
nultiple questionable points in a data segment needs further examination as 
suggested in Section IV. 
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FLOW CHART: LS7T3 BASIC PROGRAM 
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